Skip to content

[FLINK-38840][postgres] Postgres YAML connector supports emitting complete Table ID #4209

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yuxiqian merged 4 commits into from
Jan 7, 2026
Merged

[FLINK-38840][postgres] Postgres YAML connector supports emitting complete Table ID #4209

yuxiqian merged 4 commits into from
Jan 7, 2026
+188 −58

Conversation

@Mrart
Copy link
Contributor

@Mrart Mrart commented Jan 4, 2026

Currently, Postgres YAML connector maps (schema, table) as Pipeline event Table ID, and the database part is missing.

We may add a new option to generate complete Table ID (database, schema, table). This could be useful when records are written into Kafka as JSON messages.

@Mrart Mrart marked this pull request as draft January 4, 2026 00:50
@github-actions github-actions bot added postgres-cdc-connector postgres-pipeline-connector docs Improvements or additions to documentation labels Jan 4, 2026
@Mrart Mrart marked this pull request as ready for review January 4, 2026 09:47
@Mrart
Copy link
Contributor Author

Mrart commented Jan 4, 2026

@yuxiqian PTAL

@Mrart
[FLINK-38840][postgres] Postgres YAML connector supports emitting com…
00e3b60 
…plete Table ID

[FLINK-38840][postgres] Postgres YAML connector supports emitting complete Table ID

[FLINK-38840][postgres] Postgres YAML connector supports emitting complete Table ID

[FLINK-38840][postgres] Postgres YAML connector supports emitting complete Table ID
@loserwang1024
Copy link
Contributor

loserwang1024 commented Jan 5, 2026

@Mrart
It seems that even we don't have table-id.include-database option but directly emitting complete Table ID. The compability of currently pipeline won't be broke. And kafka sink can add a new option to determine whether to print database. @yuxiqian , WDYT?

@yuxiqian
Copy link
Member

yuxiqian commented Jan 5, 2026

@Mrart It seems that even we don't have table-id.include-database option but directly emitting complete Table ID. The compability of currently pipeline won't be broke. And kafka sink can add a new option to determine whether to print database. @yuxiqian , WDYT?

I'm a little concerned about changing default Table ID format. It may affect transform & route table matching rules and schema registry states where Table IDs are used as key.

@Mrart
Copy link
Contributor Author

Mrart commented Jan 5, 2026

@Mrart It seems that even we don't have table-id.include-database option but directly emitting complete Table ID. The compability of currently pipeline won't be broke. And kafka sink can add a new option to determine whether to print database. @yuxiqian , WDYT?

We still need to control the output of the full tableID through configuration to ensure that it is compatible with production

yuxiqian
yuxiqian approved these changes Jan 5, 2026
View reviewed changes
Copy link
Member

@yuxiqian yuxiqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM. Would @loserwang1024 like to take another look?

@Mrart
Copy link
Contributor Author

Mrart commented Jan 6, 2026

@loserwang1024 PTLK

@loserwang1024
Copy link
Contributor

loserwang1024 commented Jan 6, 2026

LGTM

@yuxiqian yuxiqian merged commit 99c00f3 into apache:master Jan 7, 2026
62 of 63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yuxiqian yuxiqian yuxiqian approved these changes

Assignees

No one assigned

Labels

approved docs Improvements or additions to documentation postgres-cdc-connector postgres-pipeline-connector reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Mrart @loserwang1024 @yuxiqian